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English learners (ELs) bring valuable cultural and linguistic 
knowledge and experiences with them to school. Yet these 
students face the unique challenge of simultaneously learn- 
ing English and other academic content, which can lead to 
lower scores in reading and writing in English compared to 
their non-EL peers.' Despite the additional challenges faced 
by ELs, teachers receive little training in how to effectively 
teach this population of students.” 


The Pathway to Academic Success Project trains teachers to 
improve the reading and writing abilities of ELs who have 

an intermediate level of English proficiency by incorporating 
cognitive strategies into reading and writing instruction. 

The cognitive strategies include goal setting, tapping prior 


knowledge, asking questions, making predictions, 
articulating and revising understanding of text, and 
evaluating writing.* 


This What Works Clearinghouse (WWC) intervention report, 
part of the WWC’s English Learners topic area, explores the 
effects of the Pathway to Academic Success Project on writing 
quality, writing conventions, and literacy achievement 

for ELs. The WWC identified five studies of the Pathway to 
Academic Success Project. Three of these studies meet WWC 
standards. The evidence presented in this report is from 
these three studies on ELs—with a sample in which 94% of 
students are Hispanic—in grades 6 to 12 in three urban and 
three suburban school districts. 


What Happens When English Learners Participate in the Pathway to Academic 


Success Project?* 


The evidence indicates that implementing the Pathway 
to Academic Success Project has potentially positive 
effects on writing quality, writing conventions, and 
literacy achievement, compared with business-as-usual 
professional development and English language arts 
instruction. 


Findings on the Pathway to Academic Success Project from three 
studies that meet WWC standards are shown in Table 1. The 
table reports an effectiveness rating, an improvement index, 
and the number of studies and students that contributed to 
the findings. The effectiveness rating is based on the quality 
of the designs used in studies, whether the findings are favor- 
able or unfavorable for the intervention, and the number 

of studies that tested the intervention. See Box 1 for more 
information on interpreting effectiveness ratings. 


In order to help readers judge the practical importance of 
an intervention’s effect, the WWC translates findings across 


studies into an “improvement index” by averaging find- 
ings that meet WWC standards within the same outcome 
domain. The improvement index can be interpreted as the 
expected change in percentile rank for an average compar- 
ison group student if that student had received the inter- 
vention. For example, an improvement index of +16 means 
that the expected percentile rank of the average comparison 
group student would increase by 16 points if the student 
received instruction from a Pathway to Academic Success 
Project teacher instead of the instruction provided to the 
comparison group. A positive improvement index does not 
necessarily mean the estimated effect is statistically signif- 
icant. Results for each individual outcome measure within 
domains are shown in Table 4. 


The evidence presented in this report is based on available 
research. Findings and conclusions could change as new 
research becomes available. 


Table 1. Summary of findings on the Pathway to Academic Success Project from studies that meet 


WWC standards 


Study findings 


Evidence meeting WWC standards (version 4.0) 


Outcome domain Effectiveness rating 


Improvement index 


(percentile points) Number of studies Number of students 
Writing quality Potentially positive effects +16 2 444 
Writing conventions Potentially positive effects +3 1 2,024 
Literacy achievement Potentially positive effects +3 1 2,726 


Note: For more information about outcome measures, see study descriptions in Tables 6, 8, and 10. The effects of the Pathway to Academic Success Project are not known 
for other outcomes within the English Learners topic area, including alphabetics, reading fluency, reading comprehension, writing productivity, English language proficiency, 


mathematics achievement, science achievement, and social studies achievement. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the English Learners topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 
effectiveness ratings and the statements used in this report: 


Effectiveness Rating Rating interpretation 


Description of the evidence 


Positive (or negative) effects 
an outcome 


The intervention is likely to change 


Strong evidence of a positive (or negative) 
effect, with no overriding contrary evidence 


Potentially positive (or negative) effects 


The intervention may change an outcome 


Evidence of a positive (or negative) effect 
with no overriding contrary evidence 


No discernible effects 


The intervention may result in little 
to no change in an outcome 


No affirmative evidence of effects 


Mixed effects 
effects on an outcome 


The intervention has inconsistent 


Evidence includes studies in at least two of 
these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


How Is the Pathway to Academic Success Project Implemented? 


The following section provides details of how districts and 
schools can implement the Pathway to Academic Success 
Project intervention. This information can help educators 
identify the requirements for implementing the intervention 
and determine whether implementation would be feasible 
in their districts or schools. Information presented in this 
section comes from the studies that meet WWC standards 
(Kim et al., 2011; Olson et al., 2016, 2020) and from 
correspondence with the intervention developer. 


* Goal: The Pathway to Academic Success Project trains 
teachers to improve the reading and writing abilities of 
ELs who have at least an intermediate level of English 
proficiency by incorporating cognitive strategies into 
reading and writing instruction. 


¢ Target population: The Pathway to Academic Success 
Project is designed for grades 6 to 12 general education 
English language arts (ELA) teachers who have ELs in 
their classrooms. The target student population is ELs in 
general education classrooms who have an intermediate 
level of English proficiency. Students with an intermediate 
level of English proficiency are proficient in colloquial 
English, but are still building skills in academic language, 
reading, and writing in English. 


¢ Method of delivery: Teachers receive the Pathway 
to Academic Success Project training through 
professional development and coaching offered by 
experienced Pathway to Academic Success Project 
teachers. Teachers then implement the strategies 
during their regular ELA class time. 


Comparison condition: In the three studies that 
contribute to this intervention report, students in the 
comparison group were taught by teachers who did not 


receive Pathway to Academic Success Project training. 
Teachers received the regular professional development 
offered by their schools or school districts. 


Frequency and duration of service: Teachers 
participate in 46 hours of professional development per 
year, over the course of 2 years. This training includes 
five or six full-day meetings and five after-school 
meetings throughout each school year. Teachers are also 
encouraged to meet in professional learning communities 
to discuss implementation for at least 1 hour each week. 
The instructional time teachers spend working with 
students on the Pathway to Academic Success Project 
strategies varies from teacher to teacher. However, most 
teachers report spending at least 60 hours of their class 
time using Pathway to Academic Success Project strategies 
with students during each school year. 


Intervention components: The key components of the 
Pathway to Academic Success Project, which are teacher 
professional development, coaching, and instructional 
materials and supplies, are described in Table 2. 


Table 2. Components of the Pathway to Academic Success Project 


Key component 


Description 


Teacher professional 
development 


Coaching 


Instructional materials 
and supplies 


The Pathway to Academic Success Project includes 2 years of professional development sessions focused on training in 
literary response and analysis, comprehension and analysis of informational nonfiction texts, synthesis of multiple texts, 

and development of clear, coherent, focused essays. Teachers are trained to use a cognitive strategies toolkit that includes 
reading and writing strategies such as setting goals, tapping prior knowledge, asking questions, making predictions, 
interpreting texts, articulating and revising understanding of text, and evaluating and revising their writing. Teachers are 

also provided a curricular approach and instructional strategies for demonstrating the cognitive strategies in the toolkit to 
students, including direct instruction, modeling, and guided practice. Teachers are also trained on how to use a student 
writing assessment to determine individual student needs and strengths and then provide tailored writing instruction. Districts 
may also opt to have the Pathway to Academic Success Project identify a guest speaker to present to teachers during the 
professional development sessions. 


Teachers receive ongoing support on how to integrate cognitive strategies into their schools’ English language arts (ELA) 
curriculum from coaches who are typically experienced Pathway to Academic Success Project teachers. Coaches may be 
teachers selected by district leadership or retired National Writing Project teachers who are trained to coach by the Pathway 
to Academic Success Project. Coaches conduct informal visits to participating classrooms and provide written feedback to 
teachers. They also attend the professional development with teachers and assist teachers in implementing the Pathway to 
Academic Success Project cognitive strategies approach into their standard ELA curricula. District literacy specialists also 
provide support by serving as liaisons between Pathway to Academic Success Project staff and district and school staff. 


Throughout the school year, teachers receive classroom materials that they can use to enhance their instruction of cognitive 
strategies, such as wall posters with visuals representing cognitive strategies, preprinted class sets of readings and handouts, 
and bookmarks and lists of cognitive sentence starters for students. Teachers also receive materials for helping students 
apply cognitive strategies to revise their pretest writing assessment into a finished essay, as well as supplies for students to 
implement the strategies such as highlighters and binders. Finally, teachers receive model lessons to use in their classrooms. 
These model lessons are based on texts that are culturally relevant to students and that teachers have used during 
professional development. 


What Does the Pathway to Academic Success Project Cost? 


This preliminary list of costs is not designed to be 
exhaustive; rather, it provides educators an overview of 
the major resources needed to implement the Pathway to 
Academic Success Project. The program costs described in 
Table 3 reflect the costs for an individual school district to 
participate in the Pathway to Academic Success Project and 
are based on information available as of June 2021. The 
developer reported that the typical cost including both 
required and optional cost ingredients in Table 3 is $65 per 


Table 3. Cost ingredients for Pathway to Academic Success Project 


student directly served by the Pathway to Academic Success 
Project, assuming 175 students per teacher in a district with 
8 schools and 8 teachers participating in each school. The 
total cost per district in this scenario is $728,000 for both 
years. Without the optional costs, the per-student cost is 
$33.38 and cost per district in this scenario is $373,800 

for both years. The cost will vary based on the number 

of teachers and students served and whether the district 
includes the optional elements described in Table 3. 


Cost 
ingredients Description Source of funding 
Personnel Teachers receive in-person training during 10 full-day meetings and 10 after-school meetings over — School districts or schools 
the course of 2 school years. A full-day meeting costs $4,500 and an after-school meeting costs pay for personnel costs as 
$1,500. The cost includes the trainer, and the trainer is typically a Pathway to Academic Success part of Pathway to Academic 
Project staff member. However, the intervention developer may provide training to district staff so Success Project. 
they can independently deliver Pathway to Academic Success Project training. The cost of the 
training is the same but allows the district to deliver its own training in the future. 
Pathway to Academic Success Project coaches receive a $1,500 stipend each year and usually 
serve three to six teachers. District literacy specialists receive a $2,500 stipend each year. Districts 
may also opt to provide teachers a stipend for participation ($1,000 per teacher per year). 
Guest speakers at trainings are optional and cost $500 to $2,000, depending on the speaker. 
Districts may also pay for optional essay readers to read and score student essays. Essay readers 
cost $200 per class section; costs will vary depending on the number of class periods and whether 
district staff or Pathway to Academic Success Project staff read the essays. 
Facilities Standard meeting facilities are included in the meeting costs, and schools and districts may reduce — School districts or schools pay for 
the cost of facilities by providing their own or using donated facilities. facility costs as part of Pathway 
to Academic Success Project 
or can provide their own or use 
donated facilities for training. 
Equipment Pathway to Academic Success Project materials, which are provided in the first year, typically cost — School districts or schools pay 


and materials 


$2,000 per teacher when purchased in bulk for at least 60 teachers. The materials include model 
lessons and classroom materials to support instruction, as well as supplies such as bookmarks, 
highlighters, and binders. Districts may opt to provide participating teachers with a $1,500 stipend 
across both school years to purchase a classroom library to support Pathway to Academic Success 
Project instruction. General office supplies and copying cost an additional $6,400 per year. 


Pathway to Academic Success Project staff also customize materials for the after-school meetings 
to address a participating district's specific needs; for example, designing a session on adapting 
the cognitive strategies approach for the district's adopted textbook. Customizing the after-school 
meeting materials costs an additional $5,000 per year ($1,000 per meeting). 


for equipment and materials as 
part of Pathway to Academic 
Success Project. 


For More Information: 


About the Pathway to Academic Success Project 
University of California, Irvine Education Building, 401 East Peltason, Suite 3100, Irvine, CA 92697 
Attn: Dr. Carol Booth Olson 
Email: cbolson@uci.edu Web: http://www.ucipathwayproject.com Phone: (949) 824-7842 (x5922) 
About the cost of the intervention 
The developer provided information about the Pathway to Academic Success Project costs. 


Research Summary 


The WWC identified five studies that investigated the 
effectiveness of the Pathway to Academic Success Project 
(Figure 1): 


* 1study meets WWC standards without reservations 
e 2studies meet WWC standards with reservations 
e 2studies do not meet WWC standards 


The WWC reviews findings on the intervention’s effects on 
eligible outcome domains from the three studies that meet 
standards, either with or without reservations. Based on this 
review, the WWC generates an effectiveness rating, which 
summarizes how the intervention impacts, or changes, a 
particular outcome domain. 


The three studies of the Pathway to Academic Success Project 
that meet WWC standards reported findings on writing 
quality, writing conventions, and literacy achievement. No 
other studies or findings meet WWC standards within any 
outcome domain included in the English Learners topic 
area.° Citations for the five studies reviewed for this report 
are listed in the References section, which begins on page 13. 


The WWC reports additional supplemental findings, such as 
those the study authors reported for subtests or subgroups, 
on the WWC website (https://whatworks.ed.gov). These 
supplemental findings and findings from studies that do 

not meet WWC standards do not contribute to the 
effectiveness ratings. 


Figure 1. Effectiveness ratings for Pathway to Academic Success Project 
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The Pathway to Academic Success Project has potentially positive effects on writing quality 


Two studies that meet WWC standards with reservations have findings in the writing quality domain. The WWC determined one study showed 
evidence of a positive and statistically significant effect of the Pathway to Academic Success Project on writing quality (Olson et al., 2016), while the 


other study showed evidence of an indeterminate effect (Olson et al., 2020). 


The Pathway to Academic Success Project has potentially positive effects on writing conventions 


The WWC determined that one study that meets WWC standards without reservations shows evidence of a positive and statistically significant effect 
of the Pathway to Academic Success Project on use of writing conventions (Kim et al., 2011). 


The Pathway to Academic Success Project has potentially positive effects on literacy achievement 


The WWC determined that one study that meets WWC standards without reservations shows evidence of a positive and statistically significant effect 
of the Pathway to Academic Success Project on literacy achievement (Kim et al., 2011). 


Main Findings 


Table 4 shows the findings from the three studies of the a negative number favors the comparison group. A positive 
Pathway to Academic Success Project that meet WWC or negative improvement index does not necessarily mean 
standards. The table includes WWC calculations of the the estimated effect is statistically significant. 


performance of the intervention group relative to the 
comparison group in terms of the mean difference and 
effect size. The effect size is a standardized measure of the 
effect of an intervention on outcomes, representing the 
average change expected for all individuals who are given 
the intervention (measured in standard deviations of the 
outcome measure). For the mean difference and effect size 
values, a positive number favors the intervention group and 


Based on findings from the three studies that meet WWC 
standards, the effectiveness rating for writing quality, 
writing conventions, and literacy achievement is potentially 
positive effects. The findings in the writing quality domain 
are based on 444 students. The findings in the writing 
conventions and literacy achievement domains are based 
on 2,721 and 2,726 students, respectively. 


Table 4. Findings by outcome domain from studies of the Pathway to Academic Success Project that meet 
WWC standards 


Mean . 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 

Measure (Study) sample size group group difference size index p-value 

Researcher- English learner 233 5.55 4.40 1.15 0.77 +28 <.01 

developed Academic students in (1.56) (1.52) 

Writing Assessment grades 7-12 

(Olson et al., 2016)? 

Analytic Writing English learner 211 3.31 3.21 0.10 0.03 +4 >.05 

Continuum for students in (1.11) (1.17) 

Literary Analysis grades 7-12 

(Olson et al., 2020)° 

Outcome average for writing quality across all studies 0.40 +16 Statistically 
significant 

California English learner 2,721 0.04 -0.05 0.09 0.09 +3 <.05 

Standards Test: students in (0.97) (1.03) 

English language grades 6-12 

arts, Writing subtest 

(Kim et al., 2011) © 

Outcome average for writing conventions (Kim et al., 2011)° 0.09 +3 Statistically 
significant 

California English learner 2,126 328.48 325.60 2.88 0.07 +3 <.05 

Standards Test: students in (41.02) (43.74) 

English language grades 6-12 

arts composite 

(Kim et al., 2011)° 

Outcome average for literacy achievement (Kim et al., 2011) 0.07 +3 Statistically 
significant 


2 The Academic Writing Assessment is a measure of analytical writing skills developed by the authors in collaboration with other researchers for the Pathway to Academic 
Success Project. 

> Intervention group means presented for this study were provided in response to an author query. The Analytic Writing Continuum for Literary Analysis is a measure of student 
writing and literary analysis skills, with a focus on developing English learners’ writing skills. 

° The California Standards Test English language arts, writing subtest is a subtest of the California Standards Test English language arts composite score. The writing subtest 
measures students’ performance on written and oral English language conventions and writing strategies. The English language arts composite score consists of the reading 
and writing subtests and measures students’ mastery of the English language arts content standards. 


In What Context Was the Pathway to Academic Success Project Studied? 


The following section provides information on the setting 
of the three studies of the Pathway to Academic Success 
Project that meet WWC standards, and a description of the 
participants in the research. This information can help 


WHERE THE STUDY WAS CONDUCTED 


educators understand the context in which the studies of 
the Pathway to Academic Success Project were conducted, 
and determine whether the program might be suitable for 
their setting. 


3 studies, 3,170 students in 71 middle and high schools in 6 districts in California 


Race 


co 


Urban and suburban districts 


Ethnicity 


94% 6% 


Other/Not specified ae 
1% 1% 


0.4% Hispanic Non-Hispanic 


Asian Native American White 


English Learner Gender 


Status: 97% 


51% Female 49% Male 


Free & Reduced-Price 
Lunch: 80% 


om HH 42 9 + FO00OCCOO=f 


Details of Each Study that Meets WWC Standards 


This section presents details for three studies of the Pathway 
to Academic Success Project that meet WWC standards. These 
details include the full study reference, findings description, 
findings summary, and description of study characteristics. 
Asummary of domain findings for each study is presented 
below, followed by a description of the study characteristics. 
These study-level details include contextual information 
about the study setting, methods, sample, intervention group, 
comparison group, outcomes, and implementation details. 


Research details for Kim et al. (2011) 


Kim, J. S., Olson, C. B., Scarcella, R., Kramer, J., Pearson, M., 
van Dyk, D., Collins, P., & Land, R. E. (2011). A randomized 
experiment of a cognitive strategies approach to text-based 


Table 5. Summary of findings from Kim et al. (2011) 


analytical writing for mainstreamed Latino English language 
learners in grades 6 to12. Journal of Research on Educational 
Effectiveness, 4(3), 231-362. https://eric.ed.gov/?id=EJ932553 


For additional information, readers should refer to the origi- 
nal studies. Findings from Kim et al. (2011) show evidence of 
a statistically significant positive effect of the Pathway to Aca- 
demic Success Project on use of writing conventions and on 
literacy achievement (Table 5).° The findings and research 
details summarized for this study come from two related 
citations, including the primary study listed above. See the 
References section, which begins on page 13, for a list of all 
related publications. 


Meets WWC Group Design Standards Without Reservations 


Study findings 
Outcome domain SaMplosizs Average Improvement Statistically 
effect size index significant 
Writing conventions 2,721 students 0.09 +3 Yes 
Literacy achievement 2,726 students 0.07 +3 Yes 


Table 6. Description of study characteristics for Kim et al. (2011) 


WwWwc 
evidence rating 


Setting 


Methods 


Study 
sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards Without Reservations. This is a randomized controlled trial with low attrition. For more 


information on how the WWC assigns study ratings, please see the WWC Procedures and Standards Handbooks (version 4.0) and 
WWC Standards Briefs, available on the WWC website. 


The study took place in 15 secondary schools (9 middle schools and 6 high schools) in the Santa Ana Unified School District, an 
urban school district in California, during the 2007-08 and 2008-09 school years. 


The authors first randomly assigned teachers within grade levels and schools to the Pathway to Academic Success Project group 
or the comparison group and then randomly assigned eligible English learners (ELs) to teachers’ general education English 
language arts (ELA) classrooms, which were a mix of ELs and native English speakers. When an intervention or comparison group 
teacher had multiple classes eligible for the study, authors selected one classroom with the highest percentage of ELs scoring at 
or above the intermediate proficiency level on the California English Language Development Test because Pathway to Academic 
Success Project materials were designed for these students. Authors randomly assigned 1,664 ELs to a Pathway to Academic 
Success Project classroom and 1,591 ELs to a comparison classroom. The sample loss after random assignment (attrition) was 
within the acceptable threshold for review: the overall attrition rate was 16% and the differential attrition rate was 3%. 


The study consisted of 52 Pathway to Academic Success Project teachers and 51 comparison group teachers. The sample included 2,726 
ELs in grades 6 to 12. For this sample, all students were ELs, 95% of students were Hispanic, and Spanish was the first language for 88% 
of students. Forty-nine percent of students were male, and 79% of students were eligible for free or reduced-price lunch. 


Pathway to Academic Success Project training occurred over 2 years, although the findings that contribute to the effectiveness 
rating in this intervention report were measured after the first year of training. During each school year, Pathway to Academic 
Success Project teachers participated in 46 total hours of training, including six full-day sessions (6 hours each) and five after- 
school sessions (2 hours each). Developers of the Pathway to Academic Success Project led the training with support from district 
literacy coaches who were experienced Pathway to Academic Success Project teachers. The first two professional development 
days focused on introducing teachers to the cognitive strategies toolkit and instructional strategies for teaching students to use the 
toolkit. Throughout the year, teachers received curriculum materials that modeled approaches to using the cognitive strategies and 
described strategies for implementing them within the schools’ ELA curricula using direct instruction, teacher modeling, and guided 
student practice. In the third and fourth professional development days, teachers focused on analyzing students’ performance 

on a writing assessment to determine strengths and areas for growth, and received further training on the implementation of 
cognitive strategies to enhance interpretive reading and analytical writing. In the fifth and sixth professional development days, 
teachers analyzed students’ post-test writing, reflected on students’ growth in writing, and made plans for year 2. Throughout 
implementation, Pathway to Academic Success Project teachers had access to the district literacy coaches who supported them in 
integrating into the ELA curriculum a cognitive strategies approach to writing. Pathway to Academic Success Project teachers also 
received 26 hours of business-as-usual professional development, as described below for teachers in the comparison group. 


Comparison group teachers received 26 hours of business-as-usual professional development that focused on the teacher's 
guide for using the district's ELA textbook (Holt Elements of Literature series). The textbook contained instruction in cognitive 
reading strategies and included questions throughout reading selections to aid students’ comprehension, but focused more on 
cognitive strategies in reading than writing. The professional development included sessions on interpreting test data, using test 
data to improve students’ California Standards Test scores, improving students’ summarizing strategies while reading, creating 
professional learning communities, and understanding how to teach the ELA textbook. Comparison group teachers did not receive 
coaching support, but did receive resources, such as classroom library books. 


Study authors reported findings for ELs on two measures from the California Standards Test. The California Standards Test: 
English language arts, writing subtest score was reviewed in the writing conventions domain and measures students’ performance 
on written and oral English language conventions and writing strategies. The California Standards Test: English language arts 
composite score, which consists of the reading and writing subtests, was reviewed in the literacy achievement domain and 
measures students’ mastery of the ELA content standards. Authors also reported results from the California Standards Test: 
English language arts, reading subtest, and these findings were considered supplemental in this review because they involve a 
subtest and are in the same domain of the California Standards Test: English language arts composite. Supplemental findings do 
not factor into the intervention’s rating of effectiveness but are available on the WWC website (httos://whatworks.ed.gov). 


Study authors also reported findings from the Assessment of Literacy Analysis outcome measure, but findings on this outcome 
measure do not meet WWC group design standards because there is high attrition and the analytic intervention and comparison 
groups do not satisfy the baseline equivalence requirement. 


The study presents findings for two cohorts of students, but only the first cohort findings meet WWC group design standards. 
Outcomes for the year 1 cohort of students were measured after teachers received the first year of Pathway to Academic Success 
Project training. Outcomes for the year 2 cohort of students were measured after teachers received the second, and final, year 

of Pathway to Academic Success Project training. Findings for the year 2 cohort of students did not meet WWC group design 
standards because the amount of attrition is unknown and the analytic intervention and comparison groups do not satisfy the 
baseline equivalence requirement. 


The study authors conducted observations of Pathway to Academic Success Project implementation and comparison classrooms 
at the end of the first year of implementation. Authors found no differences in the use of specific Pathway to Academic Success 
Project reading or writing activities during classroom observations between intervention and comparison classrooms. Study authors 
also surveyed Pathway to Academic Success Project and comparison group teachers about their classroom instruction. Nearly 
twice as many Pathway to Academic Success Project teachers reported devoting time to writing skills compared with comparison 
group teachers. 


Research details for Olson et al. (2016) 


Olson, C. B., Matuchniak, Ts; Chung, H.Q., Stumpf, R., & Findings from Olson et al. (2016) show evidence of a 
Farkas, G. (2016). Reducing achievement gaps in academic statistically significant positive effect of Pathway to Academic 
writing for Latinos and English learners in grades 7-12. Success Project on writing quality (Table 7). 


Journal of Educational Psychology, 109(1), 1-21. https://eric. 
ed.gov/?id=EJ1125530 


Table 7. Summary of findings from Olson et al. (2016) 


Meets WWC Group Design Standards Without Reservations 


Study findings 
Outcome domain Sembiosts Average Improvement Statistically 
effect size index significant 
Writing quality 233 students 0.77 +28 Yes 


Table 8. Description of study characteristics for Olson et al. (2016) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a randomized controlled trial with compromised random 
assignment because the analysis did not account for unequal probabilities of assignment to the intervention and comparison 
groups. Baseline equivalence of the intervention and comparison groups is established. 


The study took place in 16 secondary schools (eight middle and eight high schools) in the Anaheim Union High School District, an 
urban school district in California, during the 2012-13 and 2013-14 school years. 


The authors first randomly assigned students (both mainstreamed English learners [ELs] and native English speakers) to teachers’ 
general education English language arts (ELA) classrooms, then randomly assigned teachers to either the Pathway to Academic 
Success Project or comparison group. Teachers were stratified by school and grade for random assignment, with half of teachers 
assigned to the Pathway to Academic Success Project and half to the comparison group. In grade levels with an odd number of 
teachers, the additional teacher was assigned to the Pathway to Academic Success Project group. Finally, for both conditions, 

one class per teacher was selected for the study, except for three Pathway to Academic Success Project teachers for whom 

two classes were selected. Selected classes had the greatest percentage of ELs and students with the English proficiency skills 
needed to benefit most from the program. At the start of the study, 1,493 students were assigned to Pathway to Academic Success 
Project classes, and 1,705 students were assigned to comparison group classes. 


Ninety-five teachers participated in the study, with 49 teachers in the Pathway to Academic Success Project group and 46 teachers 
in the comparison group. Because three Pathway to Academic Success Project teachers had two classrooms participating in the 
study, a total of 52 Pathway to Academic Success Project classes and 46 comparison group classes were in the study. The findings 
reviewed in this report are based on the subsample of 233 EL students in grades 7 to 12. For this sample, all students were ELs, 
80% of the students were Hispanic, 9% were Native American, 6% were Asian, 4% were White, and less than 1% were Black. 
Fifty-three percent of students were male, and 90% of students qualified for free or reduced-price lunch. 


Pathway to Academic Success Project training generally followed the same format and topics used in Kim et al. (2011), described 
in Table 6. Teachers in Olson et al. (2016) also engaged in professional learning communities within their school to discuss how 

to implement lessons from the training in their classrooms. Throughout year 1, teachers received coaching support from a retired 
teacher with previous experience with Pathway to Academic Success Project. This teacher conducted three informal classroom 
observations and provided detailed written feedback to teachers. During year 2, a lead ELA teacher within each school provided 
coaching support. In addition to classroom observations, coaches attended professional development sessions with teachers from 
their assigned school and assisted teachers in integrating Pathway to Academic Success Project strategies into their lessons. 
Pathway to Academic Success Project teachers also received business-as-usual professional development provided by the school 
district, as described below for teachers in the comparison group. 


Comparison group teachers participated in business-as-usual professional development and used the district ELA textbook and 
novels for teaching. District professional development during years 1 and 2 included one full-day session led by district curriculum 
specialists on protocols for reviewing district benchmark assessments. In year 2, district curriculum specialists also led professional 
development on text complexity. 


Study authors reported findings for two cohorts of students. Outcomes for the year 1 cohort of students were measured after 
teachers received the first year of Pathway to Academic Success Project training. Outcomes for the year 2 cohort of students were 
measured after teachers received the second, and final, year of Pathway to Academic Success Project training. The finding that 
contributes to the effectiveness rating in this intervention report are for ELs on the Academic Writing Assessment outcome in year 2. 
This researcher-developed assessment measures students’ analytical writing skills using a timed, on-demand essay where 
students write in response to prompts based on one of two short stories. Students’ essays were scored using a rubric adapted from 
rubrics used to evaluate essays for the California High School Exit Exam, California STAR 7 Direct Writing Assessment, and the 
National Assessment of Educational Progress. This outcome was reviewed in the writing quality domain. 


The study also reported outcomes that meet WWC group design standards for other student subpopulations across both year 1 and 
year 2 where at least half the students were ELs. These outcomes are considered supplemental findings and, in addition to the findings 
for year 1 students, include findings by grade, gender, ethnicity, and EL status. Summaries of supplemental findings are available on the 
WWC website (httos://whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of effectiveness. 


The writing quality outcome for ELs in year 1 does not meet WWC group design standards because random assignment was 
compromised and the study did not establish baseline equivalency between the intervention and comparison groups. Study authors 
also reported effects in the literacy achievement domain on grade 10 students’ passing rate on the California High School Exit 
Examination in ELA. This outcome does not meet WWC group design standards because random assignment was compromised, 
and data were not available to establish baseline equivalence between the intervention and comparison groups. 


The study authors also reported outcomes for student populations that were not eligible for review because they did not include at least 
50% ELs. These include outcomes for White students, Black students, and students who were not eligible for free or reduced-price 
lunch. Ineligible outcomes and outcomes that do not meet WWC group design standards are not included in the intervention report. 


Trained observers conducted observations of Pathway to Academic Success Project implementation and comparison classrooms 
and rated implementation using the Pathway to Academic Success Project Quality Checklist. Authors found that Pathway to 
Academic Success Project teachers implemented Pathway to Academic Success Project-specific strategies and activities at 

a higher rate than comparison group teachers at the final observation in the spring of year 2. Pathway to Academic Success 
Project teachers and comparison group teachers did not differ in their implementation of these strategies and activities in earlier 
observations. Similarly, authors found differences in the extent to which students demonstrated effective use of Pathway to 
Academic Success Project strategies in the final observation, but not in earlier observations. 
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Research details for Olson et al. (2020) 


Olson, C. B., Woodworth, K., Arshan, N., Black, R., Chung, 
H.Q., D’Aoust, C., Dewar, T., Friedrich, L., Godfrey, L., Land, 
R., Matuchniak, T., Scarcella, R., & Stowell, L. (2020). The 
Pathway to Academic Success: Scaling up a text-based ana- 
lytical writing intervention for Latinos and English learners 
in secondary school. Journal of Educational Psychology, 4(1), 
701-717. https://eric.ed.gov/?id=EJ1249837 


Findings from Olson et al. (2020) show evidence of an inde- 
terminate effect of Pathway to Academic Success Project in the 
writing quality domain (Table 9). The findings and research 
details summarized for this study come from two related 
citations, including the primary study listed above. See the 
References section, which begins on page 13, for a list of all 
related publications. 


Table 9. Summary of findings from Olson et al. (2020) 


Meets WWC Group Design Standards Without Reservations 


Outcome domain Sample size 


Writing quality 211 students 


Study findings 
Average Improvement Statistically 
effect size index significant 
0.03 +4 No 


il 


Table 10. Description of study characteristics for Olson et al. (2020) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a randomized controlled trial with high attrition. Baseline equivalence 
of the intervention and comparison groups is established. 


The study took place in 40 schools serving grades 7 to 12 within four public school districts in urban and suburban areas of southern 
California during the 2014-15 and 2015-16 school years. Participating districts were associated with one of four National Writing Project 
(NWP) sites. 


The authors randomly assigned teachers recruited for the study and their associated classroom to either the Pathway to Academic 
Success Project or comparison group. Teachers were stratified by school and grade for random assignment. Before random assignment, 
authors selected one “focal class” for each teacher and collected rosters of students enrolled in that class. At the start of the study, 113 
teachers and 3,105 students were assigned to Pathway to Academic Success Project classes and 117 teachers, and 3,202 students were 
assigned to comparison group classes. 


The main finding that meets WWC group design standards was measured for students who had been in the study in year 1 and continued 
in year 2, after Pathway to Academic Success Project teachers received the full 2 years of training. When moving from year 1 to year 2, 
the authors intended that students remain in the same classroom to maintain the same study condition in year 2 as students experienced 
in year 1. However, due to school scheduling demands, many students were assigned to a classroom in a different study condition, and 

a large portion of students assigned to Pathway to Academic Success Project classes in year 1 did not experience a second year of 
instruction by a Pathway to Academic Success Project teacher. 


Because scoring writing samples is costly, authors randomly selected four students per class to score their essays and include in the 
analysis. For the students continuing in year 2, authors randomly selected four students per class from among students with both a pretest 
and posttest writing sample. The review calculates attrition from the original randomly assigned sample of students, and not from only 
those selected for scoring. Because the authors limited the sample eligible for random selection to only students with both a pre- and 
posttest score, the WWC cannot distinguish exclusions due to random selection from exclusions due to study attrition. Therefore, all 
exclusions are treated as attrition in this review. 


The study consisted of 230 secondary school English language arts or English language development teachers (113 Pathway to Academic 
Success Project teachers and 117 comparison group teachers). The findings are based on 211 students in grades 7 to 12, which the authors 
describe as including English learners and redesignated English proficient students. Eighty-nine percent of the students were Hispanic, 6% 
were Asian, 1% were White, and 4% were another race or ethnicity or missing this information. Fifty-two percent of English learner students 
were male. Just over half of the 211 students (55%) met the definition of English learners for this review, which included current English 
learners and students who were recently (no earlier than 2 years before the start of the study) reclassified as English proficient. 


Pathway to Academic Success Project training followed the same format and topics used in Kim et al. (2011), as described in Table 6 
above. In this implementation of the Pathway to Academic Success Project, NWP site directors led the professional development with 
support from study co-directors, doctoral students, or NWP teachers and consultants. Each school identified a teacher to serve as a 
coordinator and liaison between the NWP site director and the school. Pathway to Academic Success Project teachers also received 
business-as-usual professional development provided by the school district, as described below for teachers in the comparison group. 


Comparison group teachers participated in business-as-usual professional development and used the district ELA textbook and novels for 
teaching. All comparison group teachers attended a half-day professional development training on Houghton Mifflin Harcourt's Collections 
textbook series. Several districts also conducted professional development on district benchmark assessments and the new state Smarter 
Balanced Assessment Consortium test. 


The WWC identified the findings measured at the end of year 2 after teachers received the second, and final, year of Pathway to Academic 
Success Project training as the main findings that contribute to the effectiveness rating in this intervention report. These findings are 
measured for a group of students who received 2 years of the intervention, remaining in the study classrooms in both years. 


Findings for two other groups of students were reviewed as supplemental. This includes a group of students who participated in year 1 with 
outcomes measured after teachers received the first year of Pathway to Academic Success Project training and a group of students who 
joined the study classrooms in the second year of the study. For this last group, teachers had received 2 years of Pathway to Academic 
Success Project training, but students were exposed to just 1 year of a Pathway to Academic Success Project trained teacher. 


Study authors reported findings for ELs on the Analytic Writing Continuum for Literary Analysis in year 2, at the end of the Pathway 

to Academic Success Project training. The Analytic Writing Continuum for Literary Analysis measures students’ writing and literary 
analysis skills, with a focus on developing ELs’ writing skills. Students wrote an essay analyzing a literary nonfiction text in response to a 
researcher-developed prompt. Essays were then scored using the NWP-developed Analytic Writing Continuum rubric. 


The study also reports outcomes that meet WWC group design standards and are considered supplemental findings because they 
measure outcomes after just 1 year of exposure or for non-ELs. These include outcomes for ELs in year 1, outcomes for non-ELs in years 
1 and 2, and outcomes for ELs and non-ELs who joined the study in the second year. Summaries of supplemental findings are available on 
the WWC website (https://whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of effectiveness. 


The intervention developer provided support for implementation. Professional development sessions were staggered so that site directors 
could watch the intervention developer deliver the session to one site before leading that session for their sites. The site directors agreed 
to implement certain elements of the intervention with fidelity but had flexibility to adapt other elements to their site. 


Implementation fidelity was assessed based on teacher participation in professional development, the extent to which the content of the 
professional development was consistent with the program model, and annual teacher surveys about professional development and 
instructional practices. Authors found that teacher participation in professional development fell short of implementation targets but the 
content of the professional development met expectations. Three of the four study sites met the implementation target of at least 90% of 
teachers attending four of the five full-day professional development sessions, and two sites met the target of 90% of teachers attending 
at least three of the five after-school sessions. Pathway to Academic Success Project teachers reported receiving more ELA-focused 
professional development than comparison group teachers and that professional development had a greater emphasis on cognitive 
strategies. However, intervention and comparison group teachers reported spending similar amounts of instructional time on analytical 
essay writing and reading strategies. 
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Endnotes 


1 The U.S. Department of Education’s data story, Academic 
Performance and Outcomes for English Learners, notes 
that 9% of ELs in grade 4 and 5% in grade 8 were proficient 
in reading. Just 67% of ELs graduated high school in 
2015-16 (compared with 85% for non-ELs). Although 
students learning multiple languages may initially seem 
to lag in language development, they can become pro- 
ficient in multiple languages with appropriate supports 
(McCabe et al., 2013; Castro et al., 2013). 


2 Most states, with the exception of Florida and California, 
(1) have few requirements for new teachers in terms of 
training in effective instruction for ELs and (2) do not 
require credentials for teaching ELs or in-service training 
focused on ELs for teachers currently teaching ELs 
(Menken & Antunez, 2001; Quality Counts, 2009). 


3 The intervention is referred to as both the Pathway 
Project and the Pathway to Academic Success Project in the 
studies that contribute to this intervention report. The 
intervention developer confirmed that the Pathway to 
Academic Success Project is the most accurate intervention 
name. The description for this intervention comes from 
the studies that contribute to this intervention report and 
the intervention developer. The What Works Clearinghouse 
(WWC) requests that developers review the intervention 
description sections for accuracy from their perspective. 
The WWC provided the developer with the intervention 
description in June 2021 and the WWC incorporated 
feedback from the developer. Further verification of 
the accuracy of the descriptive information for this 
intervention is beyond the scope of this review. 


4 The literature search reflects documents publicly available 
by December 2019. Reviews of the studies in this report 
used the standards from the WWC Procedures and Standards 
Handbook (version 4.0) and the English Learners review 
protocol (version 4.0). 
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5 The effects of the Pathway to Academic Success Project are 
not known for other outcome domains within the English 
Learners topic area, including alphabetics, reading fluency, 
reading comprehension, writing productivity, general 
literacy achievement, mathematics achievement, science 
achievement, and social studies achievement. 


6 Previous systematic reviews of Kim et al. (2011) reported 
findings for the year 2 outcomes on the California 
Standards Test English language arts composite score, 
reading subtest, and writing subtest, and the Assessment 
of Literary Analysis year 1 and year 2 outcomes. These 
additional outcomes did not meet WWC group design 
standards for this review because the amount of attrition 
is unknown and the analytic intervention and comparison 
groups do not satisfy the baseline equivalence requirement. 
Therefore, this review focuses on the year 1 California 
Standards Test English language arts composite score 
and subtest outcomes. In previous reviews, the analytic 
intervention and comparison groups did satisfy baseline 
equivalence requirements; however, due to changes in the 
WWC Procedures and Standards Handbook (version 4.0), 
they did not meet the baseline equivalence requirement for 
this review. 
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